In many professional applications, a large number of video sources need to be visualized on a video wall to provide an overview of the situation. Some examples include video surveillance, complex industrial processes (e.g., oil refinery, power plant), or networks (e.g., electricity networks). In the past, due to high requirements towards delay and quality, such professional visualization systems were often implemented using dedicated hardware and networks. However, there is a growing need to incorporate off-the-shelf consumer level hardware to reduce cost, and/or increase functionality by enabling video content sharing from the video wall/control room to and from any network and device.
In chroma sub-sampling, the encoding for the color information, chroma, can be made with less resolution than that of the brightness information, luma. This method takes advantage of the fact that the perception of the human eye is less sensitive to colour differences than for differences in brightness. The notation can be YUV, where Y represents the luma/brightness component, and U and V represents the chroma/colour components. For example, for 8 bits per component, the full resolution would be 4:4:4 while 4:2:0 has reduced colour information, the latter requiring less data storage and/or bandwidth.
The video content visualized in an environment requiring high resolution, such as e.g. a control room, can comprise a combination of “natural” video from cameras and synthetic content such as graphs, maps, spread sheets, etc. Applications with synthetic content typically require very high quality. FIG. 1 illustrates the case when a YUV 4:2:0 chroma sub-sampling is implemented for synthetic screen content in the form of the word “dec”. FIG. 1a) shows the original where a part of the letter “d” with the same colour and brightness is marked with a striped pattern. FIG. 1b) shows the reconstructed image where colour and brightness of the corresponding part of the “d” has changed into a multiple of different levels, each level is indicated with a different pattern. FIG. 1c) shows the original images so that the left image of 1c) corresponds to 1a) and right imaged of 1c) corresponds to 1b).
Dedicated hardware for efficient hardware-accelerated video coding is typically available for YUV 4:2:0 profiles [1] [2]. The first version of the High Efficiency Video Coding (HEVC) [3] compression standard supports YUV 4:2:0 sub-sampling. Together these tools can yield an efficient and cheap solution for YUV 4:2:0 sub-sampling.
Hardware accelerated video coding for YUV 4:4:4 profiles are however not yet available. Thus, the support for YUV 4:4:4 sub-sampling has to be provided with non-accelerated hardware together with less efficient- and power consuming software. Further, due to the low demand, YUV 4:4:4 enabled hardware- and software implementations are expensive compared to implementations only supporting YUV 4:2:0 profiles.
Existing adaptive up-sampling techniques that use inter-component correlation between luma and chroma are proposed in [4] and [5]. These techniques are designed for natural content and do not take the specific characteristics of screen content into account (e.g. sharp discontinuous edges, homogeneous colours in nearby pixels).
The guided image filter introduced by He et al. in [6] is an edge preserving smoothing filter. Applications of this technique can be for example edge-aware smoothing, detail enhancement and HDR compression. Patent application GB2516110A discloses a method based on guided image filtering where the luma component is used as the guidance image to reconstruct the chroma components. The method depends on the covariance between the two. The guided image filter is able to transfer sharp edges from the luma component to the chroma component. When no covariance can be detected, however, the filter will merely smooth the chroma component. This is a good property for natural content, but it can introduce unwanted artifacts for synthetic screen content. FIG. 2 shows a) the input, being synthetic screen content, to a guided chroma reconstruction filter, b) the output from the same filter and c) the original input. The striped pattern in a) and b) indicates fields that are gradient-free, i.e. each striped field has the same colour and intensity. It can be seen that this field in the output image b) is decreased, thus the amount of gradient has increased which causes the edge to look less sharp.
In patent application GB2516110A the outcome can be optimized by providing additional information in the encoded stream. Also information on how to reduce the above mentioned artifacts could be included in the stream. But adding this information to the stream leads to an increased use of bandwidth.
US 2010/214472 (Tomonaga) discloses a method for improved upscaling of a graphics image (e.g. from SD to HD). Graphics content has sharper edges as natural content in the luma components. As common upscaling filters are designed for natural content, these filters will introduce new artifacts around the edges of graphics content. This introduces image degradation. By analyzing the luma histogram, the noise is detected and removed around the peaks. A luminance histogram detector is required.
EP 2 495 963 A2 (Toshiba) discloses a method for automatic conversion of video signals by detecting their type. Conversion is only done when the correct format is not present in the video signal. Video is converted to another format having the number of colour sampling points smaller than that of the correct format.
US 2008/0036792 A1 (Qualcomm) discloses an improved upsampling method by looking at the symmetry and intensity of neighboring pixels in the image. The intensity is used to adapt the variant of an interpolation filter.
US 2009/0324079 A1 (Yuan) discloses a method for using multiple filters for natural and graphics content in a single frame. It also uses motion vector information to share the context between multiple frames.
US 2011/0181778 A1 (Komaki) discloses a method for up sampling a video signal of film material while reducing the noise.